Conversation
cdcd890 to
e4cb4a7
Compare
…efix on forks A live session could only be rewound when the prompt was fully cached, so any fork that shared a long prefix but diverged near the end threw away the whole resident checkpoint and paid a full prefill. Accept a partial rewind: when the common prefix is shorter than the prompt, rewind to it and let the scheduler evaluate only the suffix, and when the whole prompt is cached keep rewinding to the last token so it is resampled. Expose the capability as ds4_engine_can_rewind() instead of testing the engine family at the call site, so the scheduler asks a question about rollback rather than about GLM. Metal DSpark is deliberately not admitted yet: ds4_session_rewind() has no DSpark path, so a rewind there would drop the checkpoint, log a rebuild, and persist a shorter snapshot than the one that was already resident. Enable it together with the engine-side snapshot reuse.
ds4_engine_can_rewind() answers one narrow question: does ds4_session_rewind() roll the engine back while keeping the checkpoint, or does it clear checkpoint_valid and force the very rebuild this helper exists to avoid? Its comment names that question and then hard-codes GLM. That was accurate when the helper was written. ds4_session_rewind() has since grown a Qwen3.8 branch that restores a verify snapshot when one matches the position and otherwise resets the graph and replays the kept transcript; both paths set state_ok = true, so the checkpoint survives and the rewind is exactly the case this gate is meant to admit. Read against today's implementation the predicate is stale, and Qwen3.8 silently loses the live prefix rewind that the rest of this PR generalises. Keep the two in step by admitting Qwen3.8 alongside GLM. DeepSeek stays out: its DSpark compressors cannot be rolled back by truncating their row counts and keep no frontier, which is the case the comment was originally about. Derived from reading ds4_session_rewind(), not from a Qwen3.8 run -- no Qwen3.8 model is available here, so the argument is that the invariant the gate states is satisfied, not that the resulting speedup was measured.
e4cb4a7 to
20cda55
Compare
|
Rebased onto The conflict. Qwen3.8 support landed on main at the exact spot where this branch adds The decision. The helper's comment states the question it answers — does So Caveat, stated plainly. That argument is read off
|
1 participant
Summary
In
ds4-server, live session memory can often be reused when a new request shares a prefix with the active session. However, under high-intensity multi-turn agent sessions (such as tool-calling workflows), we observed frequentreason=token-mismatchcache misses and massive disk spills (e.g.live kv cache miss live=165755 prompt=4977 common=4973 reason=token-mismatch).Investigation uncovered two root causes:
ds4_server.c:12371strictly checkedds4_engine_is_glm_dsa(s->engine), omitting GLM-5.3, which sharesDS4_MODEL_FAMILY_GLM_DSAbut is a distinct variant.live_prefix_rewind_target()strictly assertedif (common != prompt_len) return -1;. When a multi-turn conversation or agent tool-call branched withcommon < prompt_len(e.g. 4,973 tokens matched out of 4,977, with 4 trailing new tokens), the entire resident session of 165K tokens was rejected, triggering an expensive disk cache eviction and a full prompt re-prefill.Proposed Changes
ds4.h&ds4.c: Addedbool ds4_engine_can_rewind(ds4_engine *e)to export whether the engine supports session rewinds, so the scheduler asks a question about rollback rather than about GLM.ds4_server.c:live_prefix_rewind_target(): whencommon < prompt_len, it rewinds tocommon, keeping the shared tokens resident and requiringds4_session_sync()to only evaluate the suffix tokens. Whencommon == prompt_len, it continues to rewind toprompt_len - 1to re-evaluate the final token for logits generation.test_live_prefix_rewind_target()with regression cases covering fork rollbacks and matching the observed multi-turn token counts.Note on Metal DSpark
An earlier revision of this PR admitted Metal DSpark through the new helper. That is removed here, because
ds4_session_rewind()has no DSpark path yet:checkpoint_valid, sorewind_validfails and the request falls back to a full rebuild — the exact cost the helper is meant to avoid;kv_cache_store_current(s, slot, "evict")then runs on the already-truncated session, so the persisted snapshot is shorter than the one that was resident.Both are regressions relative to the current behaviour, where DSpark never enters the rewind branch. The DSpark branch is worth enabling together with the engine-side snapshot reuse that makes
ds4_session_rewind()able to restore its state.Verification
Machine: Apple M4, macOS 26.4.1, 16 GB. Backend: Metal.
test_live_prefix_rewind_target()covers the fork-rollback cases added here,including the observed multi-turn token counts (4973 of 4977 shared, 165755
resident). The model-backed suites (
--logprob-vectors,--long-context, ...)were not run because this machine has no GGUF checked out.